Peer Review

Harvard

2026-10-26 08:12:24

Harvard study says Claude outperformed top human reviewers across academic peer review tests

A Harvard-led study by Gary King and Steven Worthington found that Claude produced stronger peer-review reports than highly selected human experts across a head-to-head test on 124 papers. The comparison used paid reviewers from rapidPeer, a pre-submission review service tied to Harvard’s Institute for Quantitative Social Science, and Claude Opus 4.8 in both tool-free and web-enabled configurations. In blind evaluations, GPT-5.5 never chose the human review as the overall winner, while three former journal editors picked Claude 26 times out of 30. Across seven dimensions, including identifying weaknesses, evaluating methods, literature coverage and chart analysis, the AI system led in every category. The study also checked factual reliability by matching review claims against the original manuscripts. Human reviewers were judged wrong in 152 of 956 checked claims, or 15.9%, versus 57 of 1,039 claims, or 5.5%, for Claude. The authors argue that AI review may now exceed even unusually well-supported human peer review, while still leaving final publication decisions to human editors.

00

OpenAI

2026-10-11 08:05:15

OpenAI’s release of 722 math papers leaves researchers saying verification could take years

Four days after OpenAI published 722 AI-generated mathematics manuscripts on GitHub, mathematicians told The Verge that the archive appears to contain real and potentially significant results, but sorting out which claims actually hold may take years. OpenAI posted the papers on Oct. 7 and withdrew three the next day, while also noting that only 300 of 719 main results were accompanied by Lean formalization, or about 42%. The company’s own README said unformalized results may contain problems. Interviewed researchers described the release as astonishing, unprecedented and surreal. At the same time, they pointed to immediate limits: formal verification does not automatically mean the statement checked by a computer fully matches the claim made in a paper, and none of the work has gone through peer review. The update log also showed errors had already surfaced, including a sign mistake in a paper on Weil classes that led to the withdrawal of two related papers tied to the Hodge conjecture. Beyond correctness, the release is shaking career plans. Some mathematicians said grant proposals and long-running projects may have been wiped out overnight, with early-career researchers, PhD students and non-tenured academics facing the heaviest pressure.

420

Sakana AI

2026-10-10 23:53:57

Sakana AI unveils LLM-assisted peer review system that caught 73.43% of core claim errors

Sakana AI has released a research paper titled Beyond Imitation, outlining an LLM-assisted peer review system built around error detection. The system uses a Multi-Layer Review, or MLR, architecture and relies on off-the-shelf API models including Claude Sonnet 4 and Claude Haiku 3.5, without requiring GPUs or fine-tuning. Sakana AI said the cost comes to about $0.47 per review. In a benchmark containing 1,164 planted contradictions, the MLR system caught 73.43% of core claim errors after four review rounds. The best baseline model detected 14.81%. Results were weaker on real-world retracted papers, where the system posted an exact match rate of 16.11%. The paper says the framework includes three agents: an appendix agent that summarizes experimental details, a literature review agent that uses web search to locate research context, and a review agent that carries out a three-stage prompting chain. The study also says MLR focuses on different aspects than human reviewers, placing more weight on validity and experiments, while human reviewers pay more attention to clarity and novelty. The report was cited by MarkTechPost.

500

Elon Musk

2026-09-15 15:30:15

Musk calls for peer review among top AI labs before model releases

Elon Musk has called on leading artificial intelligence laboratories to peer review each other’s models before public release, according to Techub, which cited CNBC. He described the process as a way to assess model safety ahead of deployment. The proposal centers on having top AI developers examine rival systems before those models are made available publicly. Musk’s comment, as summarized in the report, frames peer review as a pre-release safety check rather than a post-launch response. No additional details on which labs would participate or how such a review process would be structured were provided in the source digest.

820

OpenAI

2026-08-09 23:25:08

OpenAI Researcher’s ‘We Don’t Read Papers’ Remark Reignites Debate Over AI Research Quality

An OpenAI researcher’s remark that “we don’t read papers anymore” has sparked fresh debate over how much trust the AI community should place in top conference publications. The discussion picked up after scrutiny of a surprisingly strong ICLR paper led online commenters to question how some headline results are produced. The broader criticism gained traction alongside a large-scale reproducibility review published in July by SAI, a group co-founded by University of Chicago computer science and data science associate professor Tan Chenhao. SAI examined all 168 oral papers from ICML 2026, a conference that received 23,918 submissions and accepted just 168 for oral presentation, or about 0.7%. According to the report, SAI completed 105 full reproductions and found that only 34 papers reproduced more than 40% of their claims, while just eight cleared the 80% mark. The group also pointed to broken code, missing files, incomplete instructions, mismatched results, and four papers that depended on models no longer available. SAI estimated the median cost of fully rerunning one ICML oral paper at about $8,900, with 17 papers costing more than $100,000 and the most expensive nearing $2.2 million. The episode has sharpened a longstanding tension: elite labs may rely less on papers in day-to-day work, but papers still remain a core gatekeeping tool for students and early-career researchers trying to enter those same institutions.

2540